{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pipeline Scoring Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a demo notebook showing how to use the `orion.analysis.analyze` function on one of the\n",
    "signals from the Demo Dataset and afterwards evaluate how well the pipeline performed.\n",
    "\n",
    "In this case, we will be processing the `S-1` signal with the `lstm_dynamic_threshold.json`\n",
    "pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Load the data\n",
    "\n",
    "In the first step, we setup the environment and load the **S-1** signal from the **Demo Dataset**,\n",
    "\n",
    "To do so, we need to import the `orion.data.load_signal` function and call it passing\n",
    "the `'S-1'` name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging;\n",
    "\n",
    "logging.basicConfig(level=logging.ERROR)\n",
    "logging.getLogger().setLevel(level=logging.ERROR)\n",
    "\n",
    "import warnings\n",
    "warnings.simplefilter(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1222819200</td>\n",
       "      <td>-0.366359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1222840800</td>\n",
       "      <td>-0.394108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1222862400</td>\n",
       "      <td>0.403625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1222884000</td>\n",
       "      <td>-0.362759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1222905600</td>\n",
       "      <td>-0.370746</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    timestamp     value\n",
       "0  1222819200 -0.366359\n",
       "1  1222840800 -0.394108\n",
       "2  1222862400  0.403625\n",
       "3  1222884000 -0.362759\n",
       "4  1222905600 -0.370746"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from orion.data import load_signal\n",
    "\n",
    "train = load_signal('S-1-train')\n",
    "test = load_signal('S-1-test')\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Detect anomalies using a pipeline\n",
    "\n",
    "Once we have the data, let us try to use the LSTM pipeline to analyze it and search for anomalies.\n",
    "\n",
    "In order to do so, we will have import the `orion.analysis.analyze` function and pass it\n",
    "the loaded data and the path to the pipeline JSON that we want to use.\n",
    "\n",
    "The output will be a ``pandas.DataFrame`` containing a table with the detected anomalies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 2054 samples, validate on 514 samples\n",
      "Epoch 1/5\n",
      "2054/2054 [==============================] - 12s 6ms/step - loss: 0.4788 - mse: 0.4788 - val_loss: 0.4413 - val_mse: 0.4413\n",
      "Epoch 2/5\n",
      "2054/2054 [==============================] - 10s 5ms/step - loss: 0.4239 - mse: 0.4239 - val_loss: 0.4340 - val_mse: 0.4340\n",
      "Epoch 3/5\n",
      "2054/2054 [==============================] - 10s 5ms/step - loss: 0.4267 - mse: 0.4267 - val_loss: 0.4313 - val_mse: 0.4313\n",
      "Epoch 4/5\n",
      "2054/2054 [==============================] - 10s 5ms/step - loss: 0.4203 - mse: 0.4203 - val_loss: 0.4298 - val_mse: 0.4298\n",
      "Epoch 5/5\n",
      "2054/2054 [==============================] - 10s 5ms/step - loss: 0.4167 - mse: 0.4167 - val_loss: 0.4240 - val_mse: 0.4240\n",
      "2568/2568 [==============================] - 4s 1ms/step\n",
      "7081/7081 [==============================] - 9s 1ms/step\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>start</th>\n",
       "      <th>end</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1398600000</td>\n",
       "      <td>1406030400</td>\n",
       "      <td>0.434961</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        start         end     score\n",
       "0  1398600000  1406030400  0.434961"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from orion.analysis import analyze\n",
    "\n",
    "hyperparameters = {\n",
    "    'keras.Sequential.LSTMTimeSeriesRegressor#1': {\n",
    "        'epochs': 5,\n",
    "        'verbose': True\n",
    "    }\n",
    "}\n",
    "\n",
    "pipeline = 'lstm_dynamic_threshold'\n",
    "\n",
    "anomalies = analyze(pipeline, train, test, hyperparameters)\n",
    "anomalies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Evaluate performance\n",
    "\n",
    "In this next step we will load some already known anomalous intervals and evaluate how\n",
    "good our anomaly detection was by comparing those with our detected intervals.\n",
    "\n",
    "For this, we will first load the known anomalies for the signal that we are using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>start</th>\n",
       "      <th>end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1398168000</td>\n",
       "      <td>1407823200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        start         end\n",
       "0  1398168000  1407823200"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from orion.data import load_anomalies\n",
    "\n",
    "known_anomalies = load_anomalies('S-1')\n",
    "known_anomalies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Afterwards, we pass the ground truth, the detected anomalies and the original data\n",
    "to the `orion.metrics.contextual_accuracy` and `orion.metrics.contextual_f1_score` functions in order\n",
    "to compute a score that indicates how good our anomaly detection was. You can read more about how to evaluate a pipeline [here](https://signals-dev.github.io/Orion/user_guides/evaluation_doc.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9859481582537517"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from orion.evaluation import contextual_accuracy, contextual_f1_score\n",
    "\n",
    "contextual_accuracy(known_anomalies, anomalies, test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8697850974171117"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "contextual_f1_score(known_anomalies, anomalies, test)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
